Crowd, the Teaching Assistant: Educational Assessment Crowdsourcing

We propose a novel educational assessment method to enhance teachers’ productivity using crowdsourcing and clustering technique. Background and Motivation Assessment of students’ work is an integral part of school teachers’ job responsibility. Various forms of assessments are done by teachers to form valid and reliable judgments of pupils’ progress for the development of the curriculum. Data from assessment has also become an integral feature of how schools and the education system as a whole are held to account for their performance. However, doing assessment constitutes a significant part of teachers workload given that assignments and examinations are routine activities in today’s education system. Assessment is a tedious job as well given that responses to the same questions need to be evaluated leading to a monotonous and repetitive work of assessment. Teachers need aid in such assessment to be able to utilize their time in creative jobs, personalized attention to students, thereby enhancing their productivity. Technology enabled solutions in the education industry have become more and more common with advancements of computer technologies and penetration of Internet. However, assessment in the teacher-student ecosystem has received little attention compared to other aspects, like making educational content available globally, creating electronic content for dissemination, developing analytic and collaboration tools. The most relevant example in assessment is Xerox IgniteTM which is a educational assessment management system1. IgniteTM is a web-based software tool for printing, scanning, and scoring hand-marked assessments that also manages the data and produces personalized reports. However, presently IgniteTM can perform automatic assessments of assignments of limited types, e.g., alphanumeric characters to lines connecting objects to other free-form marks (e.g., a circle, X, check, slash, full or partial fill-in). While it is a great leap from existing fill-in the bubble responses, Copyright c © 2013, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. http://www.acs-inc.com/health-and-human-services/ education-services/ignite-teacher-tool.aspx types of assignments covered by IgniteTM are restricted because of wide variety in types of assignments and limitations of current automatic document recognition technology. In this paper, we propose to crowdsource the educational assessment which is a novel and cost-effective mechanism that overcomes deficiencies of automatic grading, in particular lack of accuracy of hand-writing and drawing recognition. Crowdsourced education has recently been an area of interest in human computation research. (Monika 2012) presents an exploratory study on how educational organizations can leverage crowdsourcing for different educational activities. Crowdsourced methods for personalized online education, content creation, its improvement and curation, and for peer assessment are discussed in (Weld et al. 2012). The assessment mechanism is about feedback to students by their peers as part of their grading and the limitations involved therein are discussed. (Heimerl et al. 2012) presents educational assessment using Community Sourcing where they propose engaging local crowd to perform expert work (grading of non-autogradable exams) via physical kiosks. They show that community sourcing works for better accuracy at the lower cost if expert skills are available with the community, rewards are relevant to the community, and crowdsourcing kiosk is placed within the expert community. We present crowdsourced educational assessment by (i) transforming monotonous but non-autogradable assignments into micro-tasks which can be crowdsourced on popular platforms like Amazon Mechanical Turk2 (ii) designing of the tasks intelligently using clustering technique for secure distribution, and significant efficiency with respect to time and cost for crowdsourcing. Further, we propose to modify the tasks creatively to Community Source the assessment to teachers in rural and semi-urban regions for their development. Here, our aim is to promote education for a specific community and not necessarily to utilize the expertise as in (Heimerl et al. 2012). Crowdsourcing as a Solution Fig. 1(a) shows the system proposed for educational assessment using crowdsourcing. Digitization, i.e., scanning of the paper based assignments (completed by students) is the first step. We design micro-tasks in a creative way leveraging http://www.mturk.com 50 Human Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts AAAI Technical Report CR-13-01